Real-Time Handwritten Digit, Character and Mathematical Symbol Recognition Using Deep Learning

Abstract

Handwritten recognition of mathematical content is a challenging task due to diverse writing styles and the coexistence of digits, alphabets, and mathematical symbols. This project presents a unified deep learning–based system for handwritten digit, character, and mathematical symbol recognition using a multi-model architecture. The proposed framework employs Convolutional Neural Networks (CNNs) trained on standard benchmark datasets, including MNIST for digit recognition, EMNIST for handwritten character recognition, and the CROHME handwritten mathematical symbol dataset for mathematical symbol classification. The architecture combines TensorFlow/Keras-based models for digit and character recognition with a PyTorch-based optimized CNN for mathematical symbol recognition. A robust image preprocessing pipeline is designed to normalize handwritten inputs, enhance contrast, and preserve structural features critical for accurate recognition. To improve adaptability and robustness, the system incorporates a user FEEDBACK–driven correction mechanism, where misclassified samples are stored in custom datasets and matched using deep feature similarity measures during inference. This feedback-driven approach allows the system to improve recognition accuracy over time without full retraining. The system further incorporates ensemble prediction strategies and cosine similarity–based feature matching to enhance robustness.

Introduction

• The system is designed to recognize digits (0–9), alphabetic characters (A–Z, a–z), and mathematical symbols using multiple specialized neural networks.
• It uses CNNs for digit recognition, ResNet-based models for characters, and a separate CNN for mathematical symbols, improving accuracy and robustness.
• The existing system relied on traditional Artificial Neural Networks with feature extraction techniques and achieved high accuracy (~99%) under controlled conditions but had limitations.
• The proposed system significantly improves performance using deep learning and standard datasets like MNIST, EMNIST, and CROHME, achieving very high accuracy (up to ~99.64%).
• Research shows that CNN-based models outperform traditional machine learning methods in both accuracy and efficiency, with potential applications in banking, forms, and postal systems.
• The system design includes UML diagrams (use case, sequence, collaboration) to represent interactions and workflow.
• Extensive testing strategies (unit, integration, performance, stress, etc.) ensure reliability, fast predictions, and robustness against errors.
• Experimental results confirm that the system performs accurately in real-time and can handle user input effectively, including correction and retraining mechanisms.

Conclusion

The handwritten recognition system successfully delivers an accurate, efficient, and adaptive solution for recognizing handwritten digits, alphabetic characters, and mathematical symbols using deep learning techniques. By integrating a Convolutional Neural Network (CNN) trained on the MNIST dataset for digit recognition, a Residual CNN (ResNet) trained on the EMNIST ByMerge dataset for character recognition, and a dedicated CNN trained on the CROHME dataset for mathematical symbol recognition, the system effectively handles diverse handwriting styles. The real-time web-based interface enables users to draw inputs and receive instant predictions with confidence scores, while the modular backend architecture ensures scalability and maintainability.

References

[1] Mayank Jetal, “Handwritten Digit Recognition Using CNN,”doi: 10.1109/ICIPTM52218.2021.9388351, IEEE Access, 2021. [2] Saqib Aetal, “A robust CNN model for handwritten digits recognition and classification,” doi: 10.1109/AEECA49918.2020.9213530, IEEE Access, 2021. [3] Simon Cetal, “Integrating Writing Dynamics in CNN for Online Children Handwriting Recog¬nition,” doi: 10.1109/ICFHR2020.2020.00057, IEEE Access, 2020. [4] Chao Zetal, “Handwritten Digit Recognition Based on Convolutional Neural Network,” doi: 10.1109/CAC51589.2020.9326781, IEEE Access, 2021. [5] Durjoy S Metal, “CNN based common approach to handwritten character recognition of multi¬ple scripts,” doi: 10.1109/ICDAR.2015.7333916, IEEE Access, 2015. [6] Zhiqi Y; Kai F, “Design and implementation of handwritten digit recognition system based on template method,” doi: 10.1109/IAEAC.2018.8577760, IEEE Access, 2018. [7] En M Cetal, “Handwritten Character and Digit Recognition with Deep Convolutional Neural Networks: A Comparative Study,” doi: 10.1109/ICoICT58202.2023.10262721, IEEE Access, 2023. [8] Ayush K Aetal, “A Robust Model for Handwritten Digit Recognition using Machine and Deep Learning Technique,” doi: 10.1109/INCET51464.2021.9456118, IEEE Ac¬cess, 2021. [9] Caiyun M; Hong Z, “Effective handwritten digit recognition based on multi- feature extraction and deep analysis,” doi: 10.1109/FSKD.2015.7381957, IEEE Access, 2016. [10] Jie M et al, “Handwritten digits recognition based on improved label propagation algo¬rithm,” doi: 10.1109/ICCOINS.2016.7783239, IEEE Access, 2016. [11] Eshika J; Amanveer S, “Efficient Handwritten Digit Classification using Convolutional Neural Networks: A Robust Approach with Data Augmentation,” doi: 10.1109/ICACRS62842.2024.10841545, IEEE Access, 2025. [12] Subharathna N et al, “Decoding Handwritten Characters using Convolutional Neural Networks (CNNs),” doi: 10.1109/ICSCSS60660.2024.10625226, IEEE Access, 2024. [13] Chandradeep B et al, “Deep Learning for Handwritten Character Recognition,” doi: 10.1109/ICSEIET58677.2023.10303610, IEEE Access, 2023. [14] Yawei H; Huailin Z, “Handwritten digit recognition based on depth neural network,” doi: 10.1109/ICIIBMS.2017.8279710, IEEE Access, 2018. [15] Mahmoud M A G; Ashraf A Y M, “A Comparative Study on Handwriting Digit Recognition Using Neural Networks,” doi: 10.1109/ICPET.2017.20, IEEE Access, 2017. [16] Gregory Cetal, “EMNIST: Extending MNIST to handwritten letters,” doi: 10.1109/IJCNN.2017.7966217, IEEE Access, 2017. [17] Peiyu M, “Recognition of Handwritten Digit Using Convolutional Neural Network,” doi: 10.1109/CDS49703.2020.00044, IEEE Access, 2020. [18] Caihua L et al, “Handwritten character recognition with sequential convolutional neural net¬work,” doi: 10.1109/ICMLC.2013.6890483, IEEE Access, 2013. [19] Sen F, “Handwritten Digital Detection Based on Tensorflow Building SSD Model,” doi: 10.1109/ICIASE45644.2019.9074012, IEEE Access, 2020. [20] Chung-Hsing C et al, “Recognition of Handwritten English and Digits Using Stroke Features and MLP,” doi: 10.1109/SCISISIS55246.2022.10002071, IEEE Access, 2023.

Copyright

Copyright © 2026 Dr. P.V.N. Rajeswari, N. Anusha. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.